1. Field of the Invention
The present invention relates to a test system for desktop telephones and more particularly, to a system and method for the automated group testing of a plurality of telephones by a test application device such as a computer.
2. Description of the Relevant Art
Desktop telephones have internal software programs with numerous operational routines, which must operate in a trouble free manner for the end user. When designing software for a new telephone, the software engineer must debug the software before the telephones can be released to the public. When debugging the software often plural telephones (e.g. 20 telephones) are networked together. A test application device such as a computer typically controls the network and can input control signals directly to each telephone of the network.
These control signals cause the telephones to actually perform certain operations. The computer can send first control signals to a first telephone to cause the first telephone to call a second telephone. The first control signals cause the first telephone to perform as if certain keystrokes were entered into the keypad of the first telephone. Thereby, the first telephone can be made to dial the extension number of the second telephone. Then, the second telephone can answer the call, when the computer activates the speaker phone key of the second telephone. Next, the second telephone can transfer the call, when the computer making the second telephone activates a soft key for the transfer function followed by dialing an extension of a third telephone.
Such a test application computer does not cause buttons on the phones to be physically depressed. Rather, the computer causes the telephone to generate the internal signal associated with the pressing of a particular button. The software of the telephones is programmed to accept this computer input to control simulated keystrokes.
The computer may run thousands of telephone operations among the plurality of telephones in order to test various functions of the telephone in varying combinations and for varying durations. This form of comprehensive testing will almost certainly reveal bugs in the software programs running in the telephone under test, which must be addressed by a software engineer before the new telephones may be released to the public.
Testing operators are physically present during the testing. The operators watch the telephones undergoing the testing procedure to determine when a bug in the software occurs. Often one of the telephones will freeze up and need to be reset or rebooted. The reset or reboot may be performed by momentarily unplugging the telephone. Some other types of telephones may automatically reset or reboot themselves.
When a failure is observed, the testing operator makes note of the processes being run by the computer at the time of failure, i.e. the stage of the computer test program. Later, the computer program can be started at a point prior to the noted failure stage and the telephone failure can be replicated and more closely observed to pinpoint the timing and nature of the operation being performed when the failure occurred in order to troubleshoot the software program.
A drawback of the prior art testing methods is that the operator must be physically present and carefully observe the telephones. The operators cannot leave the telephones unattended and simply return three or four hours later. If the operators did so, they would most likely return to a computer running its testing program and several telephones in frozen states. The operators would have no way of knowing when the telephones froze, the operations being perform at the point of failure, and/or the sequence of simulation events being performed by the computer just prior to the event which lead to the failure. The operators would only know that sometime within the last three or four hours, a failure occurred and the test needs to be started over again and the telephones observed to determine the point of failure. In order to understand the failure situation, not only the expected responses to the applied test inputs, but also the plurality of status of lamps, LEDs, screens, and the status of buttons, must be watched over long periods of time over the plurality of telephones. These actions can be prohibitively difficult and time consuming for a plurality of complex telephones.
Watching the telephones during a test becomes tedious and stressful. Also, many bugs in the software go unnoticed by the testing operators. For example, if the bug does not result in a telephone freezing up, but a mere momentary erroneous display of data on the telephone, it is unlikely that the testing operator will notice the failure and take note of it for later software debugging.
Attempts have been made to automate the testing of software, and one such attempt is described in U.S. Pat. No. 5,781,720, entitled “Automated GUI Interface Testing”, by Parker et al. The '720 patent discloses a method for automated testing of both new and revised computer application programs which use a Graphical User Interface (GUI). Simulated user events such as keyboard or mouse actions are automatically input into the GUI interface which is displayed on a screen in accordance with a bitmap. The GUI is then monitored to observe the changes to the GUI in response to the input. The changes may be monitored by a computer utilizing a difference analysis technique.
Difference analysis refers to the process of comparing two observables and reporting the differences. The term observable refers to any information about the application or screen which can be captured automatically. Such observables include bitmaps, characters or texts. In situations where differences are expected and not significant, a filter may be specified by the programmer to automatically mask out insignificant differences. Bitmaps often contain areas where differences are considered insignificant. For example, the current date and time is the type of information which, if not masked out, would guarantee that the screens are never the same. This problem is solved by specifying a set of inclusion/exclusion regions which can be used to mask areas of the screen where differences are not considered significant. The actual method of difference analysis depends on the type of the observable. The '720 patent describes several types of difference analysis.
One type of difference analysis described in the '720 patent is bitmap difference analysis which involves comparing the pixels of two bitmaps. The analysis starts by checking the sizes of the bitmaps. If the bitmaps are not the same size, they are considered different. If they are the same size, the color of each pixel in the two bitmaps is compared until either a difference is detected or all pixels have matched. A second type of analysis described in the '720 patent is character screen difference analysis which compares the characters and the attributes of two screens. The analysis starts by checking the sizes of the screens. If the screens are not the same sizes, they are considered different. If they are the same size, the characters and their attributes are compared. A third type of difference analysis is text difference analysis which compares the contents of two text files, and if there are differences between the text files, the changes are noted.
U.S. Pat. No. 6,226,407, by Zabih et al., and entitled “Method and Apparatus for Analyzing Computer Screens” also discloses a method for quantitatively measuring differences in bitmap images on a computer screen using computer vision techniques. The bitmap images correspond to the rendered screens of two applications based on the same data source file. The '407 patent also discloses filtering unimportant areas of the screen. After rendering a series of screens for each application, the bitmap images are captured and interpreted to produce machine-readable visual attributes of the rendered screens. Corresponding attributes from each of the two applications are then compared to generate a set of differences. The attributes and differences are then processed to derive a set of grades reflecting the similarities between the rendered screens of the two applications.
While the difference analysis methods of the '720 patent and '407 patent are useful in analyzing the differences between a correct display of information and incorrect display of information on a computer screen, these prior art techniques are not adequate for software controlled telephones. Modern software controlled telephones can include a large number of buttons and secondary displays such as lamps, LEDs, and buttons. If the errors in these modern telephones are related to a mechanical input devices, such as the buttons, or the secondary displays, such as LEDs or lamps, in addition to the software errors, there is no way to conveniently correlate the error to the mechanical input device or secondary display.
Accordingly, there is a need for a testing method that can do more than simply detect the occurrence of an error by analyzing the differences between a correct display of information and incorrect display of information. More specifically, there is a need for a testing method that can identify an error associated with a mechanical button or secondary display of a telephone, and provide the test operator with the ability to step back through a plurality of error reports to pinpoint the source of the errors.